Why webscrape with R? 🌎

Web scraping broadly includes:

• getting (unstructured) data from the web, and
• bringing it into shape (e.g. cleaning it, getting it into tabular format).

Why web scrape? While some influential people consider “Data Scientist” 👩‍💻 to be the sexiest job of the 21st century (congratulations!), one of the sexiest just emerging academic disciplines is Computational Social Science (CSS). Why is that so?

• data abundance online
• social interaction online
• services track social behavior

Online data are a very promising vehicle to gather insights for you as data scientist for the common good.

BUT online data are usually meant for display, not a (clean) download!

Luckily, with R we can automate the whole pipeline of downloading, parsing, and post-processing to make our projects easily reproducible.

In general, remember, the basic workflow for scraping static webpages is the following.

Scraping static sites with rvest 🚜

Who doesn’t love Wikipedia? Let’s use this as our first, straight forward test case. Let’s take a look at the Cologne page.

Step 1. Load the packages rvest and stringr.

library(rvest)
library(stringr)
library(tidyverse)
library(here)

Step 2. Parse the page source. If encoding is necessary, you can add it in the reading html step.

parsed_url <- rvest::read_html("https://en.wikipedia.org/wiki/Cologne", 
                               encoding = "UTF-8")

Step 3. Extract information.

parsed_url |> 
  rvest::html_element(xpath = '//p[76]') |> 
  rvest::html_text()

## [1] "The Cologne carnival is one of the largest street festivals in Europe. In Cologne, the carnival season officially starts on 11 November at 11 minutes past 11 a.m. with the proclamation of the new Carnival Season, and continues until Ash Wednesday. However, the so-called \"Tolle Tage\" (crazy days) do not start until Weiberfastnacht (Women's Carnival) or, in dialect, Wieverfastelovend, the Thursday before Ash Wednesday, which is the beginning of the street carnival. Zülpicher Strasse and its surroundings, Neumarkt square, Heumarkt and all bars and pubs in the city are crowded with people in costumes dancing and drinking in the streets. Hundreds of thousands of visitors flock to Cologne during this time. Generally, around a million people celebrate in the streets on the Thursday before Ash Wednesday.[73]"

There are many ways to get the same content:

# Method 1: Semantic/content-based selection
# Finds the h3 heading with id "Carnival", then grabs the paragraph that follows it
# More robust because it relies on the content structure (headings and their IDs)
# Will still work even if Wikipedia adds/removes other content on the page
parsed_url |> 
  rvest::html_element(xpath = '//div[contains(@class, "mw-heading mw-heading3")]//h3[@id="Carnival"]/following::p') |> 
  rvest::html_text()

## [1] "The Cologne carnival is one of the largest street festivals in Europe. In Cologne, the carnival season officially starts on 11 November at 11 minutes past 11 a.m. with the proclamation of the new Carnival Season, and continues until Ash Wednesday. However, the so-called \"Tolle Tage\" (crazy days) do not start until Weiberfastnacht (Women's Carnival) or, in dialect, Wieverfastelovend, the Thursday before Ash Wednesday, which is the beginning of the street carnival. Zülpicher Strasse and its surroundings, Neumarkt square, Heumarkt and all bars and pubs in the city are crowded with people in costumes dancing and drinking in the streets. Hundreds of thousands of visitors flock to Cologne during this time. Generally, around a million people celebrate in the streets on the Thursday before Ash Wednesday.[73]"

# Method 2: Absolute path selection  
# Goes to the exact 76th paragraph in a very specific location in the HTML tree
# Super fragile - breaks if Wikipedia changes anything above this paragraph
# Like giving directions "take the 76th door" instead of "go to the room labeled Carnival"
parsed_url |> 
  rvest::html_element(xpath = "/html/body/div[2]/div/div[3]/main/div[3]/div[3]/div[1]/p[76]") |> 
  rvest::html_text()

## [1] "The Cologne carnival is one of the largest street festivals in Europe. In Cologne, the carnival season officially starts on 11 November at 11 minutes past 11 a.m. with the proclamation of the new Carnival Season, and continues until Ash Wednesday. However, the so-called \"Tolle Tage\" (crazy days) do not start until Weiberfastnacht (Women's Carnival) or, in dialect, Wieverfastelovend, the Thursday before Ash Wednesday, which is the beginning of the street carnival. Zülpicher Strasse and its surroundings, Neumarkt square, Heumarkt and all bars and pubs in the city are crowded with people in costumes dancing and drinking in the streets. Hundreds of thousands of visitors flock to Cologne during this time. Generally, around a million people celebrate in the streets on the Thursday before Ash Wednesday.[73]"

parsed_url |> 
  rvest::html_elements(xpath = '//*[@id="mw-content-text"]/div[1]/p[76]') |> 
  rvest::html_text()

## [1] "The Cologne carnival is one of the largest street festivals in Europe. In Cologne, the carnival season officially starts on 11 November at 11 minutes past 11 a.m. with the proclamation of the new Carnival Season, and continues until Ash Wednesday. However, the so-called \"Tolle Tage\" (crazy days) do not start until Weiberfastnacht (Women's Carnival) or, in dialect, Wieverfastelovend, the Thursday before Ash Wednesday, which is the beginning of the street carnival. Zülpicher Strasse and its surroundings, Neumarkt square, Heumarkt and all bars and pubs in the city are crowded with people in costumes dancing and drinking in the streets. Hundreds of thousands of visitors flock to Cologne during this time. Generally, around a million people celebrate in the streets on the Thursday before Ash Wednesday.[73]"

parsed_url |> 
  rvest::html_elements(xpath = '//p[(((count(preceding-sibling::*) + 1) = 179) and parent::*)]') |> 
  rvest::html_text()

## [1] "The Cologne carnival is one of the largest street festivals in Europe. In Cologne, the carnival season officially starts on 11 November at 11 minutes past 11 a.m. with the proclamation of the new Carnival Season, and continues until Ash Wednesday. However, the so-called \"Tolle Tage\" (crazy days) do not start until Weiberfastnacht (Women's Carnival) or, in dialect, Wieverfastelovend, the Thursday before Ash Wednesday, which is the beginning of the street carnival. Zülpicher Strasse and its surroundings, Neumarkt square, Heumarkt and all bars and pubs in the city are crowded with people in costumes dancing and drinking in the streets. Hundreds of thousands of visitors flock to Cologne during this time. Generally, around a million people celebrate in the streets on the Thursday before Ash Wednesday.[73]"

nypl_url <- "https://www.nypl.org/books-more/recommendations/best-books/adults?year=2021"
nypl100 <- rvest::read_html(nypl_url)

body_nodes <- nypl100 |> 
 rvest::html_elements("body") |> 
 rvest::html_children()

body_nodes |> 
 rvest::html_children()

# XPath
title <- nypl100 |> 
  rvest::html_elements(xpath = '//ul/li/div/div/h4') |> 
  rvest::html_text2()


# CSS
author <- nypl100 |> 
  rvest::html_elements(css = '.spbb-card__byline--grid') |> 
  rvest::html_text2()


# Combination of XPath with CSS
summary <- nypl100 |> 
  rvest::html_elements(xpath = '//*[contains(concat( " ", @class, " " ), concat( " ", "spbb-card__description--grid", " " ))]') |> 
  rvest::html_text2()


# Select elements by class name using XPath
# Finds all elements where the class attribute contains "spbb-card__description--grid"
summary <- nypl100 |> 
  rvest::html_elements(xpath = '//*[contains(@class,"spbb-card__description--grid")]') |> 
  rvest::html_text()

knitr::kable(data.frame(title, author, summary) |> head(3))

# YOUR ANSWER HERE

# load html
url_p <- rvest::read_html("https://en.wikipedia.org/wiki/List_of_Spotify_streaming_records")

# extract table
spotify_table_raw <- rvest::html_table(url_p, header = T) |> # extracts all <table> elements from page
  purrr::pluck(1) |> # get table in nth place (1)
  janitor::clean_names() # clean the table names

# inspect the table we got
head(spotify_table_raw)

## # A tibble: 6 × 6
##   rank  song                    artist_s     streams_billions release_date ref  
##   <chr> <chr>                   <chr>        <chr>            <chr>        <chr>
## 1 1     "\"Blinding Lights\""   The Weeknd   5.056            29 November… [1]  
## 2 2     "\"Shape of You\""      Ed Sheeran   4.577            6 January 2… [2]  
## 3 3     "\"Starboy\""           The Weeknd … 4.123            21 Septembe… [3]  
## 4 4     "\"Someone You Loved\"" Lewis Capal… 4.074            8 November … [4]  
## 5 5     "\"Sweater Weather\""   The Neighbo… 4.064            3 December … [5]  
## 6 6     "\"As It Was\""         Harry Styles 4.059            1 April 2022 [6]

tail(spotify_table_raw)

## # A tibble: 6 × 6
##   rank                 song         artist_s streams_billions release_date ref  
##   <chr>                <chr>        <chr>    <chr>            <chr>        <chr>
## 1 96                   "\"Clean Ba… Dream S… 2.428            29 April 20… ""   
## 2 97                   "\"Sweet Ch… Guns N'… 2.427            3 June 1988  "[96…
## 3 98                   "\"Levitati… Dua Lip… 2.424            1 October 2… "[97…
## 4 99                   "\"Creep\""  Radiohe… 2.421            21 Septembe… ""   
## 5 100                  "\"Jocelyn … XXXTent… 2.420            31 October … "[98…
## 6 As of 4 October 2025 "As of 4 Oc… As of 4… As of 4 October… As of 4 Oct… "As …

# clean up table a bit 
spotify_table <- spotify_table_raw |>
  dplyr::mutate(song = stringr::str_remove_all(song, '\"'), # remove quotation marks from song titles
                streams_billions = as.numeric(streams_billions) # make stream_billions a numeric variable
                ) |> 
  dplyr::slice(1:100) |> # drop last row
  dplyr::select(-ref) # drop ref col

head(spotify_table)

## # A tibble: 6 × 5
##   rank  song              artist_s                 streams_billions release_date
##   <chr> <chr>             <chr>                               <dbl> <chr>       
## 1 1     Blinding Lights   The Weeknd                           5.06 29 November…
## 2 2     Shape of You      Ed Sheeran                           4.58 6 January 2…
## 3 3     Starboy           The Weeknd and Daft Punk             4.12 21 Septembe…
## 4 4     Someone You Loved Lewis Capaldi                        4.07 8 November …
## 5 5     Sweater Weather   The Neighbourhood                    4.06 3 December …
## 6 6     As It Was         Harry Styles                         4.06 1 April 2022

# YOUR ANSWER HERE

Scraping multiple pages 🤖

Whenever you want to really understand what’s going on within the functions of a new R package, it is very likely that there is a relevant article published in the Journal of Statistical Software. Let’s say you are interested in how the journal was doing over the past years [2023-2024].

Step 1. Inspect the source. Basically, follow steps to extract the XPath information.

browseURL("http://www.jstatsoft.org/issue/archive")

Step 2 Develop a scraping strategy. We need a set of URLs leading to all sources. Inspect the URLs of different sources and find the pattern. Then, construct the list of URLs from scratch.

## URL list build

# base
baseurl <- "http://www.jstatsoft.org/article/view/v"

# volume number
volurl <- as.character(105:110) # volumes 105 to 110

# issue number
issueurl <- c(paste0("0", 1:9), 10:12) # 01 to 12 maximum number of issues in a volume
# (there is a more efficient way of dealing with the inconsistency, but
# let's try to keep this simple

combinations <- tidyr::expand_grid(volurl, issueurl)
#tidyr::expand_grid produces all combinations of volurl and issueurl

urls_list <- combinations |> 
  dplyr::mutate(url = paste0(baseurl, volurl, 'i', issueurl)) |>
  dplyr::pull(url)

names_for_files <- combinations |>
  dplyr::mutate(name = paste0(volurl, '_', issueurl, '.html')) |>
  dplyr::pull(name)

Step 3 Think about where you want your scraped material to be stored and create a directory.

here()
tempwd <- here::here("session-05-web-scraping/data/jstatsoftStats")
#here from here package creates a character string that represents a path to a directory

dir.create(tempwd, recursive = TRUE)
#recursive = TRUE indicates that it should create all directories along the specified path if they don't exist

setwd(tempwd)

Step 4 Download the pages. Note that we did not do this step last time, when we were only scraping one page.

folder <- paste0(tempwd, "/html_articles/")
#concatenate the tempwd path with "/html_articles/" to create another path

dir.create(folder, recursive = TRUE)

for (i in seq_along(urls_list)) {
  # only update, don't replace
  if (!file.exists(paste0(folder, names_for_files[i]))) {
    # skip article when we run into an error
    tryCatch(
      download.file(urls_list[i], destfile = paste0(folder, names_for_files[i])),
      error = function(e)
        e #the error object (denoted by e) is returned but not acted upon, meaning the error is essentially ignored
    )
    # don't kill their server --> be polite!
    Sys.sleep(runif(1, 0, 1)) #one random sleep interval of 0-1 seconds
  } 
}

While R is downloading the pages for you, you can watch it directly in the directory you defined…

Check whether it worked.

list_files <- list.files(folder, pattern = "0.*") #list of file names that match the regex
list_files_path <- list.files(folder, pattern = "0.*", full.names = TRUE) #full file paths including directory paths of matched files

length(list_files)

Yay! Apparently, we scraped the HTML pages of 60 articles.

# define output first
authors <- character()
title <- character()
datePublish <- character()

# then run the loop
for (i in seq_along(list_files_path)) {
  
   html_out <- rvest::read_html(list_files_path[i], encoding = "UTF-8")
    
  authors[i] <- html_out |> 
    rvest::html_elements(xpath = '//*[contains(concat( " ", @class, " " ), concat( " ", "authors_long", " " ))]//strong') |> 
    rvest::html_text2()
    
    
  title[i] <- html_out |> 
    rvest::html_elements(xpath = '//*[contains(concat( " ", @class, " " ), concat( " ", "page-header", " " ))]') |> 
    rvest::html_text2()
    
  datePublish[i] <- html_out |> 
    rvest::html_elements(xpath = '//*[contains(concat( " ", @class, " " ), concat( " ", "article-meta", " " ))]//*[contains(concat( " ", @class, " " ), concat( " ", "row", " " )) and (((count(preceding-sibling::*) + 1) = 2) and parent::*)]//*[contains(concat( " ", @class, " " ), concat( " ", "col-sm-8", " " ))]') |> 
    rvest::html_text2()
  
}

# inspect data
authors[1:3] 
title[1:3]
datePublish[1:3]

# create a data frame
dat <- data.frame(authors = authors, title = title, datePublish = datePublish)
head(dat)

Good scraping practice

There is a set of general rules to the game:

1. You take all the responsibility for your web scraping work.
2. Think about the nature of the data. Does it entail sensitive information? Do not collect personal data without explicit permission.
3. Take all copyrights of a country’s jurisdiction into account. If you publish data, do not commit copyright fraud.
4. If possible, stay identifiable. Stay polite. Stay friendly. Obey the scraping etiquette.
5. If in doubt, ask the author/creator/provider of data for permission—if your interest is entirely scientific, chances aren’t bad that you get data.

url <- "http://a-totally-random-website.com"

#rvest's session() creates a session object that responds to HTTP and HTML methods.
rvest_session <- rvest::session(url, 
                                httr::add_headers(
                                  `From` = "my@email.com", 
                                  `UserAgent` = R.Version()$version.string
                                  )
                                )
                
scraped_text <- rvest_session |> 
            rvest::html_elements(xpath = "p//a") |> 
            rvest::html_text()

On an API far, far away… ⭐

library(httr)
library(jsonlite)
library(xml2)
library(glue)

To get data from an API, we suggest to follow a workflow like this:

1. Read the APIs documentation!
2. Get the baseurl
3. Find out the parameters referring to the resources of interest to you
4. Create a query url from the base url and the query parameters
5. Run the GET function on the query url
6. Depending on the encoding (usually, it’s json), you will need to:

• Parse the result with the content function
• Either use jsonlite or xml2 to parse the json or xml files

Let’s have a look at an example, the Star Wars API:

baseurl <- "https://swapi.dev/api/"

query <- 'films'

httr::GET(paste0(baseurl, query)) |> # Make API call
  httr::content(as = 'text') |> # extract content as text
  jsonlite::fromJSON() |> # convert JSON data into R object (nested list)
  purrr::pluck(4) |> # extract from index inside nested list (4th place corresponding to "results" in the payload) 
  select(1:3) # select cols 1-3

##                     title episode_id
## 1              A New Hope          4
## 2 The Empire Strikes Back          5
## 3      Return of the Jedi          6
## 4      The Phantom Menace          1
## 5    Attack of the Clones          2
## 6     Revenge of the Sith          3
##                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        opening_crawl
## 1 It is a period of civil war.\r\nRebel spaceships, striking\r\nfrom a hidden base, have won\r\ntheir first victory against\r\nthe evil Galactic Empire.\r\n\r\nDuring the battle, Rebel\r\nspies managed to steal secret\r\nplans to the Empire's\r\nultimate weapon, the DEATH\r\nSTAR, an armored space\r\nstation with enough power\r\nto destroy an entire planet.\r\n\r\nPursued by the Empire's\r\nsinister agents, Princess\r\nLeia races home aboard her\r\nstarship, custodian of the\r\nstolen plans that can save her\r\npeople and restore\r\nfreedom to the galaxy....
## 2                     It is a dark time for the\r\nRebellion. Although the Death\r\nStar has been destroyed,\r\nImperial troops have driven the\r\nRebel forces from their hidden\r\nbase and pursued them across\r\nthe galaxy.\r\n\r\nEvading the dreaded Imperial\r\nStarfleet, a group of freedom\r\nfighters led by Luke Skywalker\r\nhas established a new secret\r\nbase on the remote ice world\r\nof Hoth.\r\n\r\nThe evil lord Darth Vader,\r\nobsessed with finding young\r\nSkywalker, has dispatched\r\nthousands of remote probes into\r\nthe far reaches of space....
## 3                                                 Luke Skywalker has returned to\r\nhis home planet of Tatooine in\r\nan attempt to rescue his\r\nfriend Han Solo from the\r\nclutches of the vile gangster\r\nJabba the Hutt.\r\n\r\nLittle does Luke know that the\r\nGALACTIC EMPIRE has secretly\r\nbegun construction on a new\r\narmored space station even\r\nmore powerful than the first\r\ndreaded Death Star.\r\n\r\nWhen completed, this ultimate\r\nweapon will spell certain doom\r\nfor the small band of rebels\r\nstruggling to restore freedom\r\nto the galaxy...
## 4               Turmoil has engulfed the\r\nGalactic Republic. The taxation\r\nof trade routes to outlying star\r\nsystems is in dispute.\r\n\r\nHoping to resolve the matter\r\nwith a blockade of deadly\r\nbattleships, the greedy Trade\r\nFederation has stopped all\r\nshipping to the small planet\r\nof Naboo.\r\n\r\nWhile the Congress of the\r\nRepublic endlessly debates\r\nthis alarming chain of events,\r\nthe Supreme Chancellor has\r\nsecretly dispatched two Jedi\r\nKnights, the guardians of\r\npeace and justice in the\r\ngalaxy, to settle the conflict....
## 5                   There is unrest in the Galactic\r\nSenate. Several thousand solar\r\nsystems have declared their\r\nintentions to leave the Republic.\r\n\r\nThis separatist movement,\r\nunder the leadership of the\r\nmysterious Count Dooku, has\r\nmade it difficult for the limited\r\nnumber of Jedi Knights to maintain \r\npeace and order in the galaxy.\r\n\r\nSenator Amidala, the former\r\nQueen of Naboo, is returning\r\nto the Galactic Senate to vote\r\non the critical issue of creating\r\nan ARMY OF THE REPUBLIC\r\nto assist the overwhelmed\r\nJedi....
## 6                            War! The Republic is crumbling\r\nunder attacks by the ruthless\r\nSith Lord, Count Dooku.\r\nThere are heroes on both sides.\r\nEvil is everywhere.\r\n\r\nIn a stunning move, the\r\nfiendish droid leader, General\r\nGrievous, has swept into the\r\nRepublic capital and kidnapped\r\nChancellor Palpatine, leader of\r\nthe Galactic Senate.\r\n\r\nAs the Separatist Droid Army\r\nattempts to flee the besieged\r\ncapital with their valuable\r\nhostage, two Jedi Knights lead a\r\ndesperate mission to rescue the\r\ncaptive Chancellor....

query <- 'people/?search=skywalker'

httr::GET(paste0(baseurl, query)) |> # Make API call
  httr::content(as = 'text') |> # extract content as text
  jsonlite::fromJSON() |>  #convert JSON data into R object (nested list)
  purrr::pluck(4)  # extract from index inside nested list (4th place corresponding to "results" in the payload)

##               name height    mass hair_color skin_color eye_color birth_year
## 1   Luke Skywalker    172      77      blond       fair      blue      19BBY
## 2 Anakin Skywalker    188      84      blond       fair      blue    41.9BBY
## 3   Shmi Skywalker    163 unknown      black       fair     brown      72BBY
##   gender                        homeworld
## 1   male https://swapi.dev/api/planets/1/
## 2   male https://swapi.dev/api/planets/1/
## 3 female https://swapi.dev/api/planets/1/
##                                                                                                                            films
## 1 https://swapi.dev/api/films/1/, https://swapi.dev/api/films/2/, https://swapi.dev/api/films/3/, https://swapi.dev/api/films/6/
## 2                                 https://swapi.dev/api/films/4/, https://swapi.dev/api/films/5/, https://swapi.dev/api/films/6/
## 3                                                                 https://swapi.dev/api/films/4/, https://swapi.dev/api/films/5/
##   species
## 1    NULL
## 2    NULL
## 3    NULL
##                                                                 vehicles
## 1 https://swapi.dev/api/vehicles/14/, https://swapi.dev/api/vehicles/30/
## 2 https://swapi.dev/api/vehicles/44/, https://swapi.dev/api/vehicles/46/
## 3                                                                       
##                                                                                                       starships
## 1                                      https://swapi.dev/api/starships/12/, https://swapi.dev/api/starships/22/
## 2 https://swapi.dev/api/starships/39/, https://swapi.dev/api/starships/59/, https://swapi.dev/api/starships/65/
## 3                                                                                                              
##                       created                      edited
## 1 2014-12-09T13:50:51.644000Z 2014-12-20T21:17:56.891000Z
## 2 2014-12-10T16:20:44.310000Z 2014-12-20T21:17:50.327000Z
## 3 2014-12-19T17:57:41.191000Z 2014-12-20T21:17:50.401000Z
##                                url
## 1  https://swapi.dev/api/people/1/
## 2 https://swapi.dev/api/people/11/
## 3 https://swapi.dev/api/people/43/

If you do not know the source format, the http_type function will help you out!

query <- 'starships/?search=death/?format=wookiee'

httr::GET(paste0(baseurl, query)) |>
  httr::http_type()

## [1] "application/json"

# sign up for API key: https://api.data.gov/signup/
#Sys.setenv(MY_API_KEY = "[YOUR API KEY GOES HERE]") # store your personal API key in R environment

# US Congress: Bills
baseurl <- "https://api.congress.gov/v3/" # base url (remains consistent across queries)
api_key <- glue::glue("api_key={Sys.getenv('MY_API_KEY')}") # your personal API key
query <- "bill" # query

try2 <- GET(glue::glue(baseurl, query, "?", api_key)) |> 
  content(as = "text") |> 
  fromJSON() |> 
  pluck(1) |> 
  as.data.frame()

# US Congress: Actions on a specific nomination
query <- "nomination/115/2259/actions"

GET(glue::glue(baseurl, query, "?", api_key)) |> 
  content(as = "text") |> 
  fromJSON() |> 
  pluck(1) |> 
  as.data.frame()

# US Congress: Summaries filtered by congress and bill type
query <- "summaries/117/hr?fromDateTime=2022-04-01T00:00:00Z&toDateTime=2022-04-03T00:00:00Z&sort=updateDate+desc"

GET(glue::glue("{baseurl}{query}&{api_key}")) |> # another way of using glue()
  content(as = 'text') |>
  fromJSON() |>
  pluck(3) |>
  as.data.frame()

# US Congress: Legislation sponsored by a specific congress member
query <- "member/L000174/sponsored-legislation.xml" # data can be formatted as xml too

GET(glue::glue(baseurl, query, "?", api_key),
    add_headers("X-Api-Key" = Sys.getenv("MY_API_KEY"))) |>
  content(as = "text") |>
  read_xml() |>
  xml_find_all("//sponsoredLegislation//title") |> 
  xml_text()